Variable Rate Shading with Unity3D and HTC Vive Pro Eye

This is a short tutorial about how to enable VRS feature for HMD using Unity3D. You can follow the complete instruction.

Variable Rate Shading (with Unity3D, HTC Vive Pro Eye)

Unlike per pixel shading, the variable rate shading (VRS) allows fine control of shading for each pixel region (16*16 pixel), with coarse shading, e.g., 1*2, 2*1, 2*2, 2*4, 4*2, 4*4. Besides, super sampling is also possible, e.g., 2x, 4x, 8x. Apart from foveated rendering, content adaptive shading, motion adaptive shading are under the hood of variable rate shading.

1. HTC Vive Eye Pro (NVidia VRS Foveated Rendering Plugin)

• HTC Vive Sense SDK gives access to the eye tracking capabilities of Vive Pro Eye through Unity and Unreal plugins as well as from native C.
• The Tobii XR SDK supports the VIVE Sense SDK

2. General installation

• Install SRanipal (SDK required in next section 3)
• Vive Pro Eye Setup
  • Eye Calibration

3. Pre-requisite

• Vive eye tracking SDK (you need to sign in). This also can be found under unity asset store, Steam VR plugin.
• Vive SRanipal
  • installer (VIVE_SRanipalInstaller_1.3.2.0.msi), already been installed in previous step
  • SDK (SDK-v1.3.3.0.zip) -> you need this for next section, step 2.
• Foveated Rendering plugin (from unity asset store)

4. In Unity Project Setting (3 Packages and 1 Plugin)

1. create a new project as 3D (for 3D scene, prob: import the .obj files in the asset/any folder you like)
2. import the Vive-SRanipal-Unity-Plugin.unitypackage into your Unity project, then Assets >> Import Package >> Custom Package. It will be visible in the Project >> assets >> ViveSR
3. Project Setting » XR Plugin Management (install)
   • However, OpenVR which requires for HTC Vive Eye Pro is not available at this stage, so go to the next step.
4. Go to com.valvesoftware.unity.openvr-1.1.4 and download the package. Keep it as .zip (com.valvesoftware.unity.openvr-1.1.4.tgz). Then go to windows » package manager » Add Package from tarball to your project.
   • It is under Package >> OpenVR XR Plugin
   • OpenVR will be visible now.
   • Again open the project setting >> XR Plug-in Management >> OpenVR Loader (enable)
5. Download Tobii XR SDK, and unzip (I select v3.0.178). Go to windows >> Package manager >> Add package from the disk >> package.json (you may need to agree the licence agreement).
   • important: in the same time, extend the samples and import Getting Started (and if you need any other samples, I import everything). Note: If you are using Universal RP, you need to upgrade the shaders in the scene.
   • It will be found under Package >> Tobii XR SDK
   • for test purpose only: Run the Samples >> Tobii XR SDK >> 3.0.0 >> Getting Started >> Example_GettingStarted
   • sometime you may need to open the file and find TobiiXR Initializer
   • More about the contents is available here
   • If you already has preloaded scene (skip previous step, linked to bullet point (1)
6. download the Foveated Rendering Plugin from Unity Asset Store and import in your running project
   • the modification options are available after hitting the run in Unity
   • the plugin is also available in the github repository

     **Prob: how to directly import foveated rendering plugin from the git? Is it only for Unreal Engine? However, the documentation mentioned separate Plugin available for Unity also. **

   • To the Main Camera, drag from the ViveFoveatedRendering plugin Scripts >> ViveFoveatedRendering (.cs)
   • To modify the VRS, see the document included in the plugin.

5. VRS Features

• NVIDIA VRWorks VRS
• Support Eye Tracking(for VIVE Pro Eye only
• github: https://github.com/ViveSoftware/ViveFoveatedRendering

6. Outputs

7. Variable Rate Shading (VRS)

VRS is an extension of MSAA (Multisample Antialiasing). MSAA has a fixed ratio between the rasterized resolution and the shading resolution. VRS allows the ratio to be specified per-draw call, per primitive or relative to screen position. There are 2 tiers of VRS

1. Tier 1 VRS supports draw-call and per-primitive selection of the shader sampling frequency.
2. Tier 2 VRS adds support for screen space relative selection of the shader sampling frequency.

VRS foveated rendering uses screen space frequency control and so requires Tier 2 VRS support.

8. Troubleshooting

8.1. Camera Movement Problems

1. Camera Movement: in the Example_GettingStarted, let the XR Rig Inspector Requested Tracking Mode -> Floor, and Camera Y Offset -> 0. Otherwise movement (position and rotation) is not possible. 1: Besides, you may need to enable Project Settings >> XR Plug-in Management >> Windows Mixed Reality

8.2. Variable Rate Shading

1. VRS: Add the ViveFoveatedRendering.cs script at the Main Camera in the lowest hierarchy of the camera.
2. Project Settings >> Player (scroll down) >> Configuration >> Scripting Define Symbols >> add USE_SRANIPAL

References

1. Tutorial
2. Mario Biko’s Tutorial Post
3. Vive SDKs
4. HTC Vive learning

More good reading staff on VRS

1. VRWorks - Variable Rate Shading (VRS)
2. Delivering Dynamic Foveated Rendering with NVIDIA VRSS 2
3. NVIDIA VRSS Application Form
4. NVIDIA VRWorks™ Graphics
5. MS dev blog